Open Library
Wayback CDX Shards
CDX Index shards for the Wayback Machine.
The Wayback Machine works by looking for historic URL's based on a query. This is done by searching an index of all the web objects (pages, images, etc) that have been archived over the years.
This collection holds the index used for this purpose, which is broken up into 300 pieces so they fit into items more naturally and distribute the lookup load. Each of these 300 pieces is stored in at least 2 items, and then those are also stored on the backup machines.
These shards are not publicly downloadable.
The Wayback Machine works by looking for historic URL's based on a query. This is done by searching an index of all the web objects (pages, images, etc) that have been archived over the years.
This collection holds the index used for this purpose, which is broken up into 300 pieces so they fit into items more naturally and distribute the lookup load. Each of these 300 pieces is stored in at least 2 items, and then those are also stored on the backup machines.
These shards are not publicly downloadable.
1,200
RESULTS
rss
PART OF
Internet Archive Web CrawlsMedia Type
1,200
data
Collection
1,200
Wayback CDX Shards
1,200
Web Crawls
1,200
Internet Archive Web Crawls
Fetching more results
